Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Port virtualbox scripts to VBoxManage CLI #625

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open

Conversation

stevemk14ebr
Copy link

@stevemk14ebr stevemk14ebr commented Oct 9, 2024

Ports to VBoxManage CLI, identical logic otherwise. Errors handled gracefully for the most part. Output:

stepheneckels@flarevm-build-2:~/source/repos/flare-vm$ python3 virtualbox/vbox-export-snapshots.py 
Starting operations on FLARE-VM
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} is already shut down (state: poweroff).
Restored 'FLARE-VM'
Found existing hostonlyif vboxnet0
Verified hostonly nic configuration correct
Power cycling before export...
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} is not running (state: poweroff). Starting VM...
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} started.
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} is not powered off. Shutting down VM...
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} is shut down (status: poweroff).
Power cycling done.
Exporting /usr/local/google/home/stepheneckels/EXPORTED VMS/FLARE-VM.20241009.dynamic.ova (this will take some time, go for an 🍦!)
Exported /usr/local/google/home/stepheneckels/EXPORTED VMS/FLARE-VM.20241009.dynamic.ova! 🎉
All operations on FLARE-VM successful ✅
Starting operations on FLARE-VM.full
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} is already shut down (state: poweroff).
Restored 'FLARE-VM.full'
Found existing hostonlyif vboxnet0
Changed nic1 to hostonly
Verified hostonly nic configuration correct
Power cycling before export...
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} is not running (state: poweroff). Starting VM...
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} started.
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} is not powered off. Shutting down VM...
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} is shut down (status: poweroff).
Power cycling done.
Exporting /usr/local/google/home/stepheneckels/EXPORTED VMS/FLARE-VM.20241009.full.dynamic.ova (this will take some time, go for an 🍦!)
Exported /usr/local/google/home/stepheneckels/EXPORTED VMS/FLARE-VM.20241009.full.dynamic.ova! 🎉
All operations on FLARE-VM.full successful ✅
Starting operations on FLARE-VM.EDU
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} is already shut down (state: poweroff).
Restored 'FLARE-VM.EDU'
Found existing hostonlyif vboxnet0
Changed nic1 to hostonly
Verified hostonly nic configuration correct
Power cycling before export...
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} is not running (state: poweroff). Starting VM...
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} started.
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} is not powered off. Shutting down VM...
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} is shut down (status: poweroff).
Power cycling done.
Exporting /usr/local/google/home/stepheneckels/EXPORTED VMS/FLARE-VM.20241009.EDU.ova (this will take some time, go for an 🍦!)
Exported /usr/local/google/home/stepheneckels/EXPORTED VMS/FLARE-VM.20241009.EDU.ova! 🎉
All operations on FLARE-VM.EDU successful ✅
Done. Exiting...

Copy link
Member

@Ana06 Ana06 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work @stevemk14ebr! I need to still test the code locally, but I have added some questions and improvement suggestions already. It is good to see what we can do with VBoxManage and how it allows us to remove the virtualbox dependency. The disadvantage is that it is less flexible, as it does not export everything in the API (for example, it seems it is not possible to access the max number of adapters which would allow us to write simpler code as in the previous version) and that we need to create a subprocess everytime we want to run a command. The new code using VBoxManage also looks longer and more complicated, but we may be able to simplify it a bit.

What about keeping both the version using the virtualbox library and the new one using VBoxManage until we have tested and migrated everything else?

Also, I think we need some documentation in /virtualbox/README.md.

Comment on lines +45 to +48
except subprocess.CalledProcessError as e:
# exit code is an error
print(f"Error running VBoxManage command: {e} ({e.stderr})")
raise Exception(f"Error running VBoxManage command")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is it needed to catch the exception to print and error and re-reise it? I see the same pattern in other functions as well.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style choice, this throws a pretty error to the top level main to print out. I can change if you think there's a more pythonic style

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have seen in other Python code that an exception is re-triggered to add extra details or format the exception differently, but without a print that duplicates similar information. The print apart from duplicating the information, can make the output difficult to digest in this case, as {e.stderr} is rendering the output that is likely to be the long help message from VBoxManage. I think we should remove the try-catch, as the re-triggered exception is almost the same:

  • Original exception: Command '['VBoxManage', 'list2', 'list', 'hostonlyifs']' returned non-zero exit status 2
  • Retriggered exception: Error running VBoxManage command: Command '['VBoxManage', 'list2', 'list', 'hostonlyifs']' returned non-zero exit status 2.
Suggested change
except subprocess.CalledProcessError as e:
# exit code is an error
print(f"Error running VBoxManage command: {e} ({e.stderr})")
raise Exception(f"Error running VBoxManage command")

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, if we re-raise the exception, I think we should use a more concrete typ of exception like RuntimeError.

virtualbox/vbox-export-snapshots.py Outdated Show resolved Hide resolved
virtualbox/vbox-export-snapshots.py Outdated Show resolved Hide resolved
virtualbox/vbox-export-snapshots.py Outdated Show resolved Hide resolved
virtualbox/vbox-export-snapshots.py Outdated Show resolved Hide resolved
session.unlock_machine()
print(f"Restored '{snapshot_name}' and changed its adapter(s) to host-only")

vm_uuid = get_vm_uuid(VM_NAME)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to get the UUID? It seems like the commands work with the VM_NAME (we may need to enclose the entire name in double quotes to avoid issues with spaces), or am I missing something?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could rely on the VM_NAME alone, but I use the UUID so that we can support multiple VMs of the same name and be sure we refer to the same VM consistently for all operations

virtualbox/vbox-export-snapshots.py Outdated Show resolved Hide resolved
virtualbox/vbox-export-snapshots.py Outdated Show resolved Hide resolved
virtualbox/vbox-export-snapshots.py Outdated Show resolved Hide resolved
virtualbox/vbox-export-snapshots.py Show resolved Hide resolved
@stevemk14ebr
Copy link
Author

stevemk14ebr commented Oct 10, 2024

for example, it seems it is not possible to access the max number of adapters which would allow us to write simpler code as in the previous version

we can, the vminfo command lists all 8 adapters (the max) and any unset adapters have the value 'none'. The code doesn't need to check the max adapters because it lists all of them, even if they're unset, so we always loop all 8 adapters.

What about keeping both the version using the virtualbox library and the new one using VBoxManage until we have tested and migrated everything else

I have no issues with not merging these PRs (I will send more for the other two scripts) until we are ready to drop the virtualbox package dependency entirely. I would not want to keep two version around though, that goes against the spirit of doing this work. While the code does appear more complex, the port was actually quite straightforward, there is just a lot of logic to parse the text CLI output and handle the errors nicely. Some things are different than the virtualbox package for sure, but there are not any glaring things missing from the CLI. In the long term this should be very easy to maintain as the CLI does not often change. More importantly though on some setup the python .so that virtualbox uses is not build/included, and the package is unmaintained for +1 year at this time, so we should not rely on it anymore.

@stevemk14ebr stevemk14ebr changed the title Port vbox-export-snapshots to VBoxManage CLI Port virtualbox scripts to VBoxManage CLI Oct 11, 2024
Copy link
Member

@Ana06 Ana06 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have tested vbox-export-snapshots.py and I it fails because the network interface does not have a name. I though this happened in the previous version when exporting the VM, but it seems like setting it could be the issue and because you start the VM (what I was not doing in the previous version) it fails even before exporting it:

Starting operations on FLARE-VM
VM {40138663-f254-412b-8776-10a7cc08daea} is already shut down (state: poweroff).
Restored 'FLARE-VM'
VM {40138663-f254-412b-8776-10a7cc08daea} is already shut down (state: poweroff).
Found existing hostonlyif vboxnet0
Changed nic1
Nic configuration verified correct
Power cycling before export...
VM {40138663-f254-412b-8776-10a7cc08daea} is not running (state: poweroff). Starting VM...
Error running VBoxManage command: Command '['VBoxManage', 'startvm', '{40138663-f254-412b-8776-10a7cc08daea}', '--type', 'gui']' returned non-zero exit status 1. (VBoxManage: error: Nonexistent host networking interface, name '' (VERR_INTERNAL_ERROR)
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component ConsoleWrap, interface IConsole
)
Error checking VM state: Error running VBoxManage command
Unexpectedly failed doing operations on FLARE-VM. Exiting...
Done. Exiting...

I reported what I think was a bug in https://www.virtualbox.org/ticket/22158. But what really confuses me is that it seems it does work for you. 😕

virtualbox/vbox-adapter-check.py Outdated Show resolved Hide resolved
virtualbox/vbox-adapter-check.py Outdated Show resolved Hide resolved
virtualbox/vbox-adapter-check.py Show resolved Hide resolved
virtualbox/vbox-adapter-check.py Show resolved Hide resolved
virtualbox/vbox-export-snapshots.py Outdated Show resolved Hide resolved
Comment on lines +146 to +151
hostonlyifs_output = run_vboxmanage(["list", "hostonlyifs"]) # Get the updated list
for line in hostonlyifs_output.splitlines():
if line.startswith("Name:"):
hostonlyif_name = line.split(":")[1].strip()
print(f"Created hostonlyif {hostonlyif_name}")
return
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this code is repeated twice, please move it to a function to remove the code duplication

Comment on lines +45 to +48
except subprocess.CalledProcessError as e:
# exit code is an error
print(f"Error running VBoxManage command: {e} ({e.stderr})")
raise Exception(f"Error running VBoxManage command")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have seen in other Python code that an exception is re-triggered to add extra details or format the exception differently, but without a print that duplicates similar information. The print apart from duplicating the information, can make the output difficult to digest in this case, as {e.stderr} is rendering the output that is likely to be the long help message from VBoxManage. I think we should remove the try-catch, as the re-triggered exception is almost the same:

  • Original exception: Command '['VBoxManage', 'list2', 'list', 'hostonlyifs']' returned non-zero exit status 2
  • Retriggered exception: Error running VBoxManage command: Command '['VBoxManage', 'list2', 'list', 'hostonlyifs']' returned non-zero exit status 2.
Suggested change
except subprocess.CalledProcessError as e:
# exit code is an error
print(f"Error running VBoxManage command: {e} ({e.stderr})")
raise Exception(f"Error running VBoxManage command")

Comment on lines +45 to +48
except subprocess.CalledProcessError as e:
# exit code is an error
print(f"Error running VBoxManage command: {e} ({e.stderr})")
raise Exception(f"Error running VBoxManage command")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, if we re-raise the exception, I think we should use a more concrete typ of exception like RuntimeError.

virtualbox/vbox-export-snapshots.py Show resolved Hide resolved
vm.save_settings()

# cmd is an array of string arguments to pass
def run_vboxmanage(cmd):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should extract all common functions (the ones used in several scripts) into a file we can re-use to avoid duplication.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants